Abnormality detection device, abnormality detection method, and abnormality detection system

ABSTRACT

A technique is provided for detecting the presence or absence of an abnormality with respect to an object appearing in a target image with high accuracy without using AI, even in an environment in which luminance values and colors are likely to fluctuate. An abnormality detection device (230) includes: an image input unit (232) for inputting an input image indicating an abnormality detection target object; a gradient distribution generation unit (234) for dividing the input image into predetermined regions and generating, for each region, a gradient distribution that indicates a distribution of a luminance gradient direction of the region; and an abnormality determination unit (236) for determining the presence or absence of an abnormality by analyzing the gradient distribution generated for each region.

TECHNICAL FIELD

The present disclosure relates to an abnormality detection device, anabnormality detection method, and an abnormality detection system.

BACKGROUND OF THE INVENTION

In recent years, with the progress of IT, a large number of sensors havebeen arranged throughout society, and extremely large amounts of dataare accumulated.

Under such circumstances, various measures to utilize the accumulatedimage data have been considered. In particular, as image contents suchas photographs, videos, and images increase, there is demand for atechnique for accurately detecting whether or not an abnormality existswith respect to an object appearing in an image.

Conventionally, several proposals have been made for detecting thepresence or absence of an abnormality with respect to an objectappearing in an acquired image.

CITATION LIST Non-Patent Documents

-   [Non-Patent Document 1]: “AI is the Key to Solving the Problem of    How to Inspect Transmission Lines Spanning a Total Length of    Approximately 14,000 km,” Oct. 25, 2018-   [Non-Patent Document 2]: Nikkei Business “Countermeasures for Big    Data and Aging Electric Cables in IoT” May 31, 2017-   [Non-Patent Document 3]: Ryuichi Ishino, Fujio Tsutsumi, Yoshiyuki    Ueno, “Detection of Damaged Cables Using Aerial Video for Inspection    of Transmission Lines,” Transactions of the Institute of Electrical    Engineers of Japan B, vol. 126, no. 4, pp. 407-414, July 2006.

SUMMARY OF INVENTION Technical Problem

Non-Patent Document 1 describes a technique for assisting inspectionoperations of transmission lines by analyzing images acquired by anaerial vehicle such as a helicopter using AI and identifying damage ordeterioration of the transmission lines.

Non-Patent Document 2 describes a technique for detecting abnormalitiescaused by lightning strikes or snow by using a robot that runs on top ofthe transmission lines to image capture the appearance of thetransmission lines, and analyzing the captured images with a proprietaryAI to determine portions that are not a straight line as“disconnections.”

Non-Patent Literature 3 describes a method of using image analysis todetect an abnormality in a transmission line by analyzing the deviationfrom a change in an average luminance value or the color of a normalportion.

However, since both of the above-mentioned Non-Patent Documents 1 and 2rely on AI-based image analysis, a large amount of learning data fortraining the AI is required in order to achieve good detection accuracy,and therefore, image capture and labeling by an expert worker becomecostly.

In addition, in the technique described in the above-mentionedNon-Patent Document 3, since the luminance value and color used as thebasis of determination of the abnormality vary greatly depending on theweather, time, and surrounding environment, the detection accuracy islimited, and application to environments in which the luminance valueand color are likely to vary is difficult.

Accordingly, it is an object of the present disclosure to provide atechnique for detecting the presence or absence of an abnormality withrespect to an object appearing in an analysis target image with highaccuracy without using AI, even in an environment in which luminancevalues or colors are likely to fluctuate.

Means for Solving the Problems

In order to solve the above problem, one representative abnormalitydetection device of the present disclosure includes an image input unitfor inputting an input image of an abnormality detection target object;a gradient distribution generation unit for dividing the input imageinto predetermined regions and generating, for each region, a gradientdistribution that indicates a distribution of a luminancegradient-direction of the region; and an abnormality determination unitfor determining a presence or absence of an abnormality by analyzing thegradient distribution generated for each region.

Advantageous Effects of Invention

According to the present disclosure, it is possible to provide atechnique for detecting the presence or absence of an abnormality withrespect to an object appearing in an analysis target image with highaccuracy without using AI, even in an environment in which luminancevalues or colors are likely to fluctuate.

Problems, configurations, and effects other than those described abovewill be made clear by the following description of embodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a computer system for implementing theembodiments of the present disclosure.

FIG. 2 is a diagram illustrating an example of a configuration of anabnormality detection system according to the embodiments of the presentdisclosure.

FIG. 3 is a diagram illustrating an input image showing an abnormalitydetection target object and a luminance gradient in the input imageaccording to the embodiments of the present disclosure.

FIG. 4 is a diagram illustrating an example of the notation of aluminance gradient according to the embodiments of the presentdisclosure.

FIG. 5 is a flowchart illustrating an example of the process which showsthe flow of the first half of the abnormality detection method accordingto the embodiments of the present disclosure.

FIG. 6 is a flowchart illustrating an example of the process which showsthe flow of the second half of the abnormality detection methodaccording to the embodiments of the present disclosure.

FIG. 7 is a diagram illustrating an example of a gradient distributionaccording to the embodiments of the present disclosure.

FIG. 8 is a flowchart illustrating another example of the process whichshows the flow of the second half of the abnormality detection methodaccording to the embodiments of the present disclosure.

FIG. 9 is a diagram illustrating a concept of the processing illustratedin FIG. 8 .

DESCRIPTION OF EMBODIMENT(S)

Hereinafter, a conventional example and the embodiments of the presentdisclosure will be described with reference to the drawings. It shouldbe noted that the present disclosure is not limited to theseembodiments. In addition, in the description of the drawings, the samecomponents are denoted by the same reference numerals.

BACKGROUND AND SUMMARY OF THE PRESENT DISCLOSURE

As described above, as image content such as photographs, videos, andimages increase, there is demand for techniques for accurately detectingwhether or not an abnormality exists with respect to an object(hereinafter referred to as an “abnormality detection target object”)appearing in an image.

Conventionally, in cases where visual inspection of substrates, devices,or the like are performed by image analysis at manufacturing sites orthe like, inspection accuracy is improved by image capturing theinspection target with a fixed angle of view and fixed lighting.

However, in recent years, there has been increased demand forinspections in environments where a fixed angle of view and fixedlighting conditions are guaranteed, such as at manufacturing sites. Forexample, the application of image processing is being considered for thevisual inspection of infrastructure such as transmission lines, bridges,roads, and tunnels. Although the use of robots and drones for inspectionis progressing, unlike visual inspections at factories, it is extremelydifficult to maintain fixed image capture conditions such as angle ofview and lighting.

On the other hand, although AI techniques such as machine-learning havebeen considered to reduce the effects of differences in image captureconditions, many AI techniques require high-performance computers, andin addition, it is necessary to collect large amounts of abnormal imagedata in addition to normal images.

Accordingly, in the present disclosure, abnormality determination isperformed based on a luminance gradient of an input image indicating anabnormality detection target image. Since the gradient direction of theluminance does not change even if there is a change in brightness due tochanges in the weather or the like, more robust abnormality detectionthan when using changes to luminance values or color as indices becomespossible, and there is no need to collect a large amount of learningdata for training an AI. In this way, according to the presentdisclosure, it is possible to provide a technique for detecting thepresence or absence of an abnormality with respect to an objectappearing in an analysis target image with high accuracy without usingAI, even in an environment in which luminance values or colors arelikely to fluctuate.

Here, the abnormality detection target object refers to an object to besubjected to abnormality detection, and may be any object such as asilicon wafer used for manufacturing an electronic circuit, a powertransmission line, an outer wall of a building, or the like, but as willbe described later, is preferably an object with a periodic pattern.

In addition, the term abnormality here refers to a property that isdifferent from the normal property of an abnormality detection targetobject that serves as the target object to be subjected to abnormalitydetection, and various abnormalities are conceivable depending on thetype of the abnormality detection target object. For example, in thecase that the abnormality detection target object is a powertransmission line, damage to the power transmission line, melting due tolightning strikes, disconnections, deformation, and the like can beconsidered, and in the case that the abnormality detection target objectis a substrate, cracking, chipping, warping, and the like can beconsidered.

(Hardware Configuration)

Referring first to FIG. 1 , a computer system 300 for implementing theembodiments of the present disclosure will be described. The mechanismsand devices of the various embodiments disclosed herein may be appliedto any suitable computing system. The main components of the computersystem 300 include one or more processors 302, a memory 304, a terminalinterface 312, a storage interface 314, an I/O (Input/Output) deviceinterface 316, and a network interface 318. These components may beinterconnected via a memory bus 306, an I/O bus 308, a bus interfaceunit 309, and an I/O bus interface unit 310.

The computer system 300 may include one or more general purposeprogrammable central processing units (CPUs), 302A and 302B, hereincollectively referred to as the processor 302. In some embodiments, thecomputer system 300 may contain multiple processors, and in otherembodiments, the computer system 300 may be a single CPU system. Eachprocessor 302 executes instructions stored in the memory 304 and mayinclude an on-board cache.

In some embodiments, the memory 304 may include random accesssemiconductor memory, storage device, or storage medium (either volatileor non-volatile) for storing data and programs. The memory 304 may storeall or a part of the programs, modules, and data structures that performthe functions described herein. For example, the memory 304 may store anabnormality detection application 350. In some embodiments, theabnormality detection application 350 may include instructions orstatements that execute the functions described below on the processor302.

In some embodiments, the abnormality detection application 350 may beimplemented in hardware via semiconductor devices, chips, logic gates,circuits, circuit cards, and/or other physical hardware devices in lieuof, or in addition to processor-based systems. In some embodiments, theabnormality detection application 350 may include data other thaninstructions or statements. In some embodiments, a camera, sensor, orother data input device (not shown) may be provided to communicatedirectly with the bus interface unit 309, the processor 302, or otherhardware of the computer system 300.

The computer system 300 may include a bus interface unit 309 forcommunicating between the processor 302, the memory 304, a displaysystem 324, and the I/O bus interface unit 310. The I/O bus interfaceunit 310 may be coupled with the I/O bus 308 for transferring data toand from the various I/O units. The I/O bus interface unit 310 maycommunicate with a plurality of I/O interface units 312, 314, 316, and318, also known as I/O processors (IOPs) or I/O adapters (IOAs), via theI/O bus 308.

The display system 324 may include a display controller, a displaymemory, or both. The display controller may provide video, audio, orboth types of data to the display device 326. Further, the computersystem 300 may also include a device, such as one or more sensors,configured to collect data and provide the data to the processor 302.

For example, the computer system 300 may include biometric sensors thatcollect heart rate data, stress level data, and the like, environmentalsensors that collect humidity data, temperature data, pressure data, andthe like, and motion sensors that collect acceleration data, movementdata, and the like. Other types of sensors may be used. The displaysystem 324 may be connected to a display device 326, such as a singledisplay screen, television, tablet, or portable device.

The I/O interface unit is capable of communicating with a variety ofstorage and I/O devices. For example, the terminal interface unit 312supports the attachment of a user I/O device 320, which may include useroutput devices such as a video display device, a speaker, a televisionor the like, and user input devices such as a keyboard, mouse, keypad,touchpad, trackball, buttons, light pens, or other pointing devices orthe like. A user may use the user interface to operate the user inputdevice to input input data and instructions to the user I/O device 320and the computer system 300 and receive output data from the computersystem 300. The user interface may be presented via the user I/O device320, such as displayed on a display device, played via a speaker, orprinted via a printer.

The storage interface 314 supports the attachment of one or more diskdrives or direct access storage devices 322 (which are typicallymagnetic disk drive storage devices, but may be arrays of disk drives orother storage devices configured to appear as a single disk drive). Insome embodiments, the storage device 322 may be implemented as anysecondary storage device. The contents of the memory 304 are stored inthe storage device 322 and may be read from the storage device 322 asneeded. The I/O device interface 316 may provide an interface to otherI/O devices such as printers, fax machines, and the like. The networkinterface 318 may provide a communication path so that computer system300 and other devices can communicate with each other. The communicationpath may be, for example, the network 330.

In some embodiments, the computer system 300 may be a multi-usermainframe computer system, a single user system, or a server computer orthe like that has no direct user interface and receives requests fromother computer systems (clients). In other embodiments, the computersystem 300 may be a desktop computer, a portable computer, a notebookcomputer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

Next, with reference to FIG. 2 , a configuration of an abnormalitydetection system according to the embodiments of the present disclosurewill be described.

FIG. 2 is a diagram illustrating an example of the configuration of anabnormality detection system 200 according to the embodiments of thepresent disclosure. As illustrated in FIG. 2 , the abnormality detectionsystem 200 includes a sensor device 205, a client terminal 210, and anabnormality detection device 230. In addition, the sensor device 205,the client terminal 210, and the abnormality detection device 230 areconnected to each other via a communication network 225.

The communication network 225 may be, for example, the Internet or aLocal Area Network (LAN).

The sensor device 205 is a sensor for acquiring an image that indicatesthe appearance of an abnormality detection target object that serves asthe target object to be subjected to abnormality detection. The image(hereinafter, the “input image” input to the abnormality detectiondevice) acquired by the sensor device 205 may be directly transmitted tothe abnormality detection device 230 or may be transmitted to the clientterminal 210.

The number, type, and arrangement of the sensor devices 205 may beappropriately selected depending on the abnormality detection targetobject. For example, in the case that the abnormality detection targetobject is a power transmission line, the sensor device 205 may be acamera mounted on a self-propelled robot that moves along the powertransmission line, and in the case that the abnormality detection targetobject is a silicon wafer used for manufacturing an electronic circuit,the sensor device 205 may be a camera mounted on a device that processesthe silicon wafer.

In addition, the sensor device 205 may acquire object identificationinformation for identifying the abnormality detection target object inaddition to the image indicating the appearance of the abnormalitydetection target object. Thus, for example, in the case that anabnormality is detected with respect to the abnormality detection targetobject, a user such as an operator who confirms the detected abnormalitycan easily identify the abnormality detection target object.

The client terminal 210 is a device for requesting the abnormalitydetection device 230 to perform abnormality detection processing withrespect to the abnormality detection target and confirming anabnormality notification output by the abnormality detection system 200.The client terminal 210 may be, for example, a portable terminal such asa smartphone or a tablet, or a fixed terminal such as a desktop personalcomputer. In embodiments, the client terminal 210 may be a terminal usedby a user, such as an on-site worker who confirms an abnormalitydetected with respect to the abnormality detection target object.

As an example, the client terminal 210 may, after receiving imagesacquired by the sensor device 205, select, from the received images, animage that clearly indicates the appearance of the abnormality detectiontarget object that serves as the target object to be subjected toabnormality detection, and transmit the selected image and anabnormality detection request for requesting abnormality detectionprocessing with respect to the abnormality detection target object tothe abnormality detection device 230 via the communication network 225.Subsequently, as a result of the determination by the abnormalitydetection device 230, in the case that an abnormality is determined withrespect to the abnormality detection target object, a user such as anon-site worker may use the client terminal 210 to confirm theabnormality notification transmitted from the abnormality detectiondevice 230 and perform inspection, maintenance, or the like with respectto the abnormality detection target object.

The abnormality detection device 230 is a device for performing anabnormality detection process according to the embodiments of thepresent disclosure on an input image received from the sensor device 205or the client terminal 210 and detecting the presence or absence of anabnormality. As illustrated in FIG. 2 , the abnormality detection device230 includes an image input unit 232, a pre-processing unit 233, agradient distribution generation unit 234, an abnormality determinationunit 236, an output unit 238, and a storage unit 240.

The image input unit 232 is a functional unit for inputting an inputimage indicating an abnormality detection target object. For example,the image input unit 232 may receive, from the sensor device 205 or theclient terminal 210, an input image indicating the appearance of anabnormality detection target object that serves as the target object tobe subjected to abnormality detection via the communication network 225and input the input image.

The pre-processing unit 233 is a functional unit for performingpre-processing (area extraction and gray scale conversion) with respectto the input image received by the image input unit 232 in order tofacilitate the abnormality detection processing according to theembodiments of the present disclosure.

The details of the pre-processing performed by the pre-processing unit233 will be described later with reference to FIG. 5 .

The gradient distribution generation unit 234 is a functional unit thatdivides an input image into regions of a predetermined size andgenerates, for each region, a gradient distribution that indicates thedistribution of a luminance gradient direction of the region. It shouldbe noted that, in the present disclosure, the gradient distributionrepresents a distribution in the gradient direction of the luminance,and includes various forms of expression such as maps and diagrams. Aswill be described later, by using the gradient distribution generatedhere, it is possible to detect the presence or absence of an abnormalitywith respect to an object appearing in a target with high accuracywithout using AI, even in an environment in which luminance values andcolors easily fluctuate.

The abnormality determination unit 236 is a functional unit fordetermining the presence or absence of an abnormality by analyzing thegradient distribution generated for each region.

Here, for example, the abnormality determination unit 236 may detect,from among each of the gradient distributions generated for each of theregions, a gradient direction that satisfies a predetermined frequencycriterion as a distribution peak, and determine the presence or absenceof an abnormality based on the detected distribution peak and apredetermined abnormality threshold set in advance.

In addition, in an embodiment, the abnormality determination unit maygenerate an average gradient distribution by averaging each gradientdistribution generated for each of the regions, detect, in the averagegradient distribution, a gradient direction that satisfies apredetermined frequency criterion as a distribution peak, and determinethe presence or absence of an abnormality based on the detecteddistribution peak and a predetermined abnormality threshold set inadvance.

Further, in an embodiment, the abnormality determination unit mayextract, from among each of the gradient distributions generated foreach of the regions, a first gradient distribution corresponding to afirst region and a second gradient distribution corresponding to asecond region adjacent to the first region, determine a distributiondivergence between the first gradient distribution and the secondgradient distribution by comparing the first gradient distribution andthe second gradient distribution, and determine the presence or absenceof an abnormality based on the determined distribution divergence and apredetermined distribution divergence threshold set in advance.

It should be noted that the details of the processing of the abnormalitydetermination unit 236 will be described later.

The output unit 238 is a functional unit for outputting a result of theabnormality determination by the abnormality determination unit. Forexample, the output unit 238 may output, in a case that an abnormalityis determined by the determination of the presence or absence of anabnormality by the abnormality determination unit, an abnormalitynotification that includes an image indicating the region determined tobe abnormal. The abnormality notification output here may be transmittedto the client terminal 210 via the communication network 225. Asdescribed above, a user, such as an on-site worker, may use the clientterminal 210 to confirm the abnormality notification transmitted fromthe abnormality detection device 230 and perform inspection,maintenance, or the like with respect to the abnormality detectiontarget object.

The storage unit 240 is a storage device for storing the various typesof information used by the abnormality detection device 230. The storageunit 240 may store, for example, information of input images receivedfrom the sensor device 205 or the client terminal 210, information ofabnormality notifications, and the like.

The storage unit 240 may be, for example, a storage device such as ahard disk drive or a solid state drive mounted in the abnormalitydetection device 230, or may be a cloud-type storage area accessiblefrom the abnormality detection device 230.

According to the abnormality detection system 200 configured asdescribed above, it is possible to provide a technique for detecting thepresence or absence of an abnormality with respect to an objectappearing in an analysis target image with high accuracy without usingAI, even in an environment in which luminance values or colors arelikely to fluctuate.

Next, with reference to FIG. 3 to FIG. 4 , the luminance gradient usedin the abnormality detection technique according to the embodiments ofthe present disclosure will be described.

FIG. 3 is a diagram illustrating an input image 360 showing anabnormality detection target object and a luminance gradient direction365 in the input image 360, according to the embodiments of the presentdisclosure. In addition, the enlarged view 361 illustrated in FIG. 3 isan enlarged view of the predetermined region 362 in the input image 360.

The input image 360 is, for example, an image that was image captured bythe sensor device 205 described with reference to FIG. 2 and thatillustrates the appearance of a power transmission line. In addition, aswill be described later, the input image 360 illustrated in FIG. 3 is animage that has already been subjected to pre-processing (areaextraction, grayscale conversion, contrast correction, and areadivision) to facilitate the abnormality detection technique according tothe embodiments of the present disclosure.

Each pixel in the input image 360 has a luminance value of 0 to 255 as avalue representing the luminance (brightness) of that pixel. A pixelcloser to black has a lower luminance value, and a pixel closer to whitehas a higher luminance value (0=black, 255=white). In the presentdisclosure, the direction extending from higher luminance values tolower luminance values is referred to as a “gradient of the luminance”or a “luminance gradient.” In the enlarged view 361, the luminancegradient direction 365 of the block is illustrated for each of aplurality of blocks constituting the predetermined region 362 in theinput image 360.

It should be noted that, in the present disclosure, it is assumed that auniform, periodic pattern exists in the appearance of the abnormalitydetection target object, and the regions 362 here are regions having asize in which at least one period of the pattern is contained.

As shown in the notation reference 450 in FIG. 4 , in the presentdisclosure, the gradient of the luminance is set such that thehorizontal axis direction of the input image is 0 degrees, thecounterclockwise direction represents positive values, and the clockwisedirection represents negative values. According to this notationreference, the angle of the luminance gradient for each of a pluralityof blocks constituting a specific region in an arbitrary input image canbe represented using a consistent notation.

As described above, in the abnormality detection technique according tothe embodiments of the present disclosure, the abnormality in theabnormality detection target object is determined based on the gradientof the luminance of an input image indicating the abnormality detectiontarget object. By analyzing the distribution of the gradient of theluminance in an input image indicating the abnormality detection targetobject, it is possible to ascertain whether an abnormality appears inthe appearance of the abnormality detection target object. For example,in a case where the distribution of the luminance gradient for each of aplurality of blocks constituting a specific region in the input image isdispersed over various angles, or in a case where the distributionbetween adjacent regions diverges greatly, it is determined that anabnormality such as damage or deterioration exists in the appearance ofthe abnormality detection target object.

Since the direction of the gradient of the luminance does not changeeven if there is a change in brightness in the image capture environmentdue to, for example, changes in weather or the like, more robustabnormality detection than when using changes to luminance values orcolor as indices becomes possible, and there is no need to collect alarge amount of learning data for training an AI. In this way, accordingto the present disclosure, it is possible to provide a technique fordetecting the presence or absence of an abnormality with respect to anobject appearing in an analysis target image with high accuracy withoutusing AI, even in an environment in which luminance values or colors arelikely to fluctuate.

Next, with reference to FIG. 5 , a description will be given of aprocess which shows the flow of the first half of the abnormalitydetection method according to the embodiments of the present disclosure.

FIG. 5 is a flowchart illustrating an example of a process 500 whichshows the flow of the first half of the abnormality detection methodaccording to the embodiments of the present disclosure. The process 500illustrated in FIG. 5 is, for example, a process performed by the imageinput unit 232, the pre-processing unit 233, and the gradientdistribution generation unit 234 in the abnormality detection device 230illustrated in FIG. 2 .

First, in Step S510, the image input unit (for example, the image inputunit 232 illustrated in FIG. 2 ) inputs an input image. As describedabove, in this case, the image input unit may receive, from the sensordevice or the client via the communication network, an input image thatindicates the appearance of an abnormality detection target object thatserves as the target object to be subjected to abnormality detection andinput the received image. The input image here may be, for example, aRGB image.

Next, in Step S520, the pre-processing unit (for example, thepre-processing unit 233 illustrated in FIG. 2 ) may extract, from theinput image received in Step S510, an area indicating the appearance ofthe abnormality detection target image. Here, for example, in the casethat the angle of view of a sensor device such as a camera is fixed, thepre-processing unit may extract an area indicating the appearance of theabnormality detection target object based on a preset set in advancewith respect to the sensor device such as the camera. In addition, thepre-processing unit may extract an area using an existing backgroundremoval technique, a segmentation unit, or an object recognition unit.

Next, in Step S530, the pre-processing unit performs grayscaleconversion on the area of the input image extracted in Step S520. Here,the pre-processing unit may use any existing technique, and is notparticularly limited in the present disclosure.

It should be noted that, although a case in which a color image isconverted into gray scale is described as an example in the presentdisclosure, the present disclosure is not limited to this, and in a casein which there are features in the color information, the gradientanalysis to be described later can be performed on each color channel ofan RGB image.

Next, in Step S540, the pre-processing unit performs contrast-correctionon the image subjected to the grayscale conversion in Step S530. Thiscontrast correction is a process performed to enhance the visibility ofthe image by enhancing the brightness and darkness of the image andmaking the shading clearer, and is not essential in the gradientanalysis described later. Accordingly, Step S540 here may be skipped incases where computing resources are limited, for example. However, it isdesirable to perform the contrast correction because it makes it easierto visually confirm the results of the abnormality determination.

Next, in Step S550, the gradient distribution generation unit (forexample, the gradient distribution generation unit 234 illustrated inFIG. 2 ) analyzes the gradient. The gradient analysis may be performedby conventional methods such as Sobel filters, for example, and is ananalysis for expressing the size relation of pixel values betweenneighboring pixels.

According to the process 500 described above, since the luminancegradient analysis processing is performed on an image subjected to thepre-processing such as area extraction and grayscale conversion, ahigh-quality abnormality determination result can be obtained.

Next, with reference to FIG. 6 and FIG. 7 , a description will be givenof a process which shows the flow of the second half of the abnormalitydetection method according to the embodiments of the present disclosure.

FIG. 6 is a flowchart illustrating an example of a process 600 whichshows the flow of the second half of the abnormality detection methodaccording to the embodiments of the present disclosure. The process 600illustrated in FIG. 6 is a process of determining the presence orabsence of an abnormality based on the distribution peak of theluminance gradient of the input image indicating the abnormalitydetection target object, and is, for example, a process performedfollowing Step S550 of the process 500 in the first half of theabnormality detection process illustrated in FIG. 5 . In addition, FIG.7 is a diagram illustrating an example of a gradient distributionaccording to the embodiments of the present disclosure.

It should be noted that the process 600 illustrated in FIG. 6 is, forexample, a process performed by the gradient distribution generationunit 234, the abnormality determination unit 236, and the output unit238 of the abnormality detection device 230 illustrated in FIG. 2 .

First, in Step S610, the gradient distribution generation unit dividesthe input image into a plurality of regions, and generates, for eachregion, a gradient distribution that indicates the distribution of theluminance gradient direction of the region. Here, the gradientdistribution is a data structure that indicates the frequency (number oftimes) distribution at which a predetermined gradient angle appears in aparticular region, and may be represented as a histogram such as thegradient distribution 710 shown in FIG. 7 that shows the frequency perbin corresponding to different gradient angles in the range of −180degrees to 180 degrees. In this way, a number of histograms equal to thenumber of regions can be acquired, such as the set of gradientdistributions 720 shown in FIG. 7 .

As an example, as illustrated in Equations 1 and 2 below, an input imageis divided into N regions, and for each region

n  [Equation 1]

a histogram

h _(n) =[h _(n1) ,h _(n2) , . . . ,h _(nB)]^(T)  [Equation 2]

indicating the distribution of the luminance gradient direction of theregion may be generated.

Here, T means transpose and B is the number of bins in the histogram.

In addition, here, the size of the region must be such as to include oneor more periods of the design of the abnormality detection target objectwithin the region. By including several periods within the region, thecharacteristics of the normal state of the abnormality detection targetcan be represented by the gradient distribution.

It should be noted that if the size of the region is too large, itbecomes difficult to detect abnormalities in the case that there aresmall abnormalities, and abnormality detection becomes difficult. Inaddition, in the case that the gradient distribution is represented as ahistogram, the width of the bins of the histogram can be arbitrarilyset, but if the width of the bins is too small, susceptibility todistortions of the sensor devices such as cameras, lighting conditions,and the tolerance of the abnormality detection target object increases,and the characteristics of the normal state become less apparent. On theother hand, if the width of the bin is too large, the difference betweenthe normal and abnormal features becomes ambiguous. In view of theabove, it is important to appropriately set the size of the region andthe width of the bin in order to obtain a high quality abnormalitydetermination result.

Next, in Step S620, the abnormality determination unit detects thedistribution peak in the gradient distribution. Here, the distributionpeak refers to a gradient angle having the highest frequency in thegradient distribution, and is a feature for determining whether or notan abnormality exists in the abnormality detection target.

The distribution peak in the gradient distribution can be obtained byvarious methods, and is not particularly limited here. As an example, agradient angle that satisfies a predetermined frequency criterion may beidentified as the distribution peak from among the respective gradientdistributions generated for each region. The predetermined frequencycriterion may be a predetermined frequency set arbitrarily, or may be acriterion for specifying the gradient angle that has the highestfrequency or the like (that is, to find the gradient direction that isthe peak of the histogram for each region, and take the one with thehighest frequency (number of occurrences) among the peak gradientdirections as the peak).

As another example of detecting the distribution peak in the gradientdistribution, an average gradient distribution may be generated byaveraging the respective gradient distributions generated for eachregion, and a gradient that satisfies a predetermined frequencycriterion may be detected as the distribution peak in the averagegradient distribution.

For example, when the gradient distribution is expressed as a histogram,as shown in the following Equations 3-6, an average histogram

h=[h ₁ , . . . ,h _(i) , . . . h _(B)]^(T)  [Equation 3]

obtained by averaging the histograms of each region may be used todetect the bin

b  [Equation 4]

that corresponds to the peak.

$\begin{matrix}{\overset{\_}{h} = {\frac{1}{N}{\sum}_{n = 1}^{N}h_{n}}} & \lbrack {{Equation}5} \rbrack\end{matrix}$ $\begin{matrix}{b = {\arg\limits_{i}{\max( \overset{\_}{h} )}}} & \lbrack {{Equation}6} \rbrack\end{matrix}$

Next, in Step S630, the abnormality determination unit calculates anabnormality degree of each region based on the values corresponding tothe gradients of the peaks detected in Step S620. Here, the abnormalitydegree of each region is a value indicating a divergence degree(distribution divergence degree) of the gradient distribution of aspecific region from the normal distribution. As a method of calculatingthe abnormality degree, for example, a feature value graph g obtained bythe following Equations 7 and 8 may be used.

g=max(f _(max))−f _(max)  [Equation 7]

f _(max) =[h _(b1) ,h _(b2) , . . . ,h _(bN)]^(T)  [Equation 8]

Next, in Step S640, the abnormality determination unit determines thepresence or absence of an abnormality with respect to each region of theinput image based on the abnormality degree calculated based on thedistribution peak in Step S630 and a predetermined abnormality thresholdset in advance. Here, the abnormality threshold value is a valuedefining the boundary line between normal and abnormal, and may be setby a user, or may be automatically determined by the abnormalitydetermination unit based on previous abnormality determination data. Asan example, in the case that the abnormality degree calculated for aspecific region is a value equal to or greater than the predeterminedabnormality threshold value set in advance (that is, when the value ofthe feature value graph g is equal to or greater than the abnormalitythreshold value), the abnormality determination unit determines that anabnormality exists with respect to this region, and in the case that theabnormality degree calculated for the specific region is a value lessthan the predetermined abnormality threshold value set in advance, theabnormality determination unit determines that an abnormality does notexist in the region.

Next, in Step S650, the output unit (for example, the output unit 238illustrated in FIG. 2 ) outputs an abnormality notification indicatingthe result of the abnormality determination in Step S640. Here, theabnormality notification may be an image indicating the abnormalitydegree calculated for each region in the input image. In an embodiment,the abnormality notification may include an image that indicates theregions determined to be abnormal. In addition, in some embodiments, theabnormality notification may include a heat map 730 that aggregates theset of gradient distributions 720 illustrated in FIG. 7 . In the heatmap 730, the abnormality degree of each region in the input image isindicated by a color gradation.

In addition, as described above, the abnormality notification generatedin Step S650 may be transmitted to the client terminal via thecommunication network.

According to the process 600 described above, it is possible to detectthe presence or absence of an abnormality with respect to theabnormality detection target object with high accuracy based on thedistribution peak of the luminance gradient of an input image indicatingthe abnormality detection target object.

Next, with reference to FIG. 8 and FIG. 9 , another example of theabnormality determination process according to the embodiments of thepresent disclosure will be described.

FIG. 8 is a flowchart illustrating an example of a process 800 whichshows the flow in the second half of the abnormality detection methodaccording to the embodiments of the present disclosure. The process 800illustrated in FIG. 8 is a process for detecting an abnormality using amethod different from that of the process 600 illustrated in FIG. 6 ,and is a process performed following Step S550 in the process 500illustrated in FIG. 5 , for example, similarly to the process 600illustrated in FIG. 6 . FIG. 9 is a diagram illustrating a concept ofthe process 800 illustrated in FIG. 8 .

More particularly, the process 800 illustrated in FIG. 8 is a process ofdetermining the presence or absence of an abnormality by comparinggradient distributions between adjacent regions in an input image thatindicates an abnormality detection target object, and is performed by,for example, the gradient distribution generation unit 234, theabnormality determination unit 236, and the output unit 238 illustratedin FIG. 2 .

First, in Step S810, the gradient distribution generation unit dividesthe input image into a plurality of regions, and generates, for eachregion, a gradient distribution that indicates the distribution of theluminance gradient direction of the region. As described herein, here,the gradient distribution is a data structure that indicates thefrequency (number of times) distribution at which a predeterminedgradient angle appears in a particular region, and may be represented asa histogram such as the gradient distribution 710 shown in FIG. 7 thatshows the frequency per bin corresponding to different gradient anglesin the range of −180 degrees to 180 degrees. In this way, a number ofhistograms equal to the number of regions can be acquired, such as theset of gradient distributions 720 shown in FIG. 7 .

It should be noted that as Step S810 in the process 800 is substantiallythe same as Step S610 in the process 600 described with reference toFIG. 6 , the description thereof is omitted here.

Next, in Step S820, the abnormality determination unit extracts agradient distribution corresponding to adjacent regions (hereinafterreferred to as “adjacent regions”) from among the respective gradientdistributions generated for each region. For example, the abnormalitydetermination unit may extract a first gradient distributioncorresponding to a first region and a second gradient distributioncorresponding to a second region adjacent to the first region.

Next, in Step S830, the abnormality determination unit compares thegradient distributions of the adjacent regions extracted in Step S830,and determines the distribution divergence degree of these gradientdistributions. Here, the distribution divergence degree here is a valuethat indicates the distance of the gradient distributions, such that alarger distribution divergence degree of the gradient directiondistribution indicates a larger difference from the gradient directiondistribution of other regions. For example, as illustrated in FIG. 9 ,from among the set of gradient distributions 910, the gradientdistribution 911 and the gradient distribution 912 are compared, and thegradient distribution 913 and the gradient distribution 914 arecompared.

Hereinafter, a case will be considered in which the gradient directiondistributions (histogram or the like)

h _(n)  [Equation 9]

and

h _(n+1)  [Equation 10]

between adjacent regions are compared with each other, and thedistribution divergence degree is calculated.

First, as illustrated in the following Equations 11-12, the gradientdirection distribution (a histogram or the like)

h _(n)  [Equation 11]

can be represented as a probability distribution

P _(n) =h _(n)/(WH)  [Equation 12]

Here,

W  [Equation 13]

represents the width of the region (in terms of the number of pixels orthe number of blocks), and

H  [Equation 14]

represents the height of the area (in terms of the number of pixels orthe number of blocks).

Then, the Kullback-Leibler divergence (KL Divergence) D can be obtainedby Equation 15 as the distribution divergence degree.

$\begin{matrix}{D = {\sum{P_{n}\log\frac{P_{n}}{P_{n + 1}}}}} & \lbrack {{Equation}15} \rbrack\end{matrix}$

It should be noted that here, the use of adjacent regions as thesubjects of comparison is because there is a high possibility thatdistortions of the camera, lighting conditions, and the like are thesame in adjacent regions, and any combination of regions may be used aslong as it is a comparison between regions in which the appearancepatterns are estimated to be the same in normal conditions.

Next, in Step S840, the abnormality determination unit determines thepresence or absence of an abnormality with respect to each region of theinput image based on the distribution divergence degree determined inStep S830 and a predetermined distribution divergence threshold set inadvance. Here, the distribution divergence threshold value is a valuedefining the boundary line between a normal distribution divergencedegree and an abnormal distribution divergence degree, and may be set bya user, or may be automatically determined by the abnormalitydetermination unit based on previous abnormality determination data. Asan example, in the case that the distribution divergence degreecalculated for specific adjacent regions is a value equal to or greaterthan the predetermined distribution divergence threshold value set inadvance, the abnormality determination unit determines that anabnormality exists with respect to these adjacent regions, and in thecase that the distribution divergence degree calculated for the specificadjacent regions is a value less than the predetermined distributiondivergence threshold value set in advance, the abnormality determinationunit determines that an abnormality does not exist in the adjacentregions.

Next, in Step S850, the output unit (for example, the output unit 238illustrated in FIG. 2 ) outputs an abnormality notification indicatingthe results of the abnormality determination in Step S840.

It should be noted that Step S850 in the process 800 is substantiallythe same as Step S650 in the process 600 described with reference toFIG. 6 , and therefore the explanation thereof is omitted here.

According to the process 800 described above, it is possible to detectthe presence or absence of an abnormality in the abnormality detectiontarget object with high accuracy based on the distribution peak of theluminance gradient of an input image indicating the abnormalitydetection target object.

Although the embodiments of the present disclosure have been describedabove, the present disclosure is not limited to the above-describedembodiments, and various modifications can be made without departingfrom the spirit of the present disclosure.

REFERENCE SIGNS LIST

-   -   200: Abnormality detection system, 205: Sensor device, 210:        Client terminal, 225: Communication network, 230: Abnormality        detection device, 232: Image input unit, 233: Pre-processing        unit, 234: Gradient distribution generation unit, 236:        Abnormality determination unit, 238: Output unit, 240: Storage        unit

1. An abnormality detection device, the abnormality detection devicecomprising: an image input unit for inputting an input image of anabnormality detection target object; a gradient distribution generationunit for dividing the input image into predetermined regions andgenerating, for each region, a gradient distribution that indicates adistribution of a luminance gradient direction of the region; and anabnormality determination unit for determining a presence or absence ofan abnormality by analyzing the gradient distribution generated for eachregion.
 2. The abnormality detection device according to claim 1,wherein the abnormality determination unit: detects, from among each ofthe gradient distributions generated for each of the regions, a gradientdirection that satisfies a predetermined frequency criterion as adistribution peak; and determines the presence or absence of anabnormality based on the distribution peak that was detected and apredetermined abnormality threshold set in advance.
 3. The abnormalitydetection device according to claim 1, wherein the abnormalitydetermination unit: generates an average gradient distribution byaveraging each gradient distribution generated for each of the regions;detects, in the average gradient distribution, a gradient direction thatsatisfies a predetermined frequency criterion as a distribution peak;and determines the presence or absence of an abnormality based on thedistribution peak that was detected and a predetermined abnormalitythreshold set in advance.
 4. The abnormality detection device accordingto claim 1, wherein the abnormality determination unit: extracts, fromamong each of the gradient distributions generated for each of theregions, a first gradient distribution corresponding to a first regionand a second gradient distribution corresponding to a second regionadjacent to the first region; determines a distribution divergencebetween the first gradient distribution and the second gradientdistribution by comparing the first gradient distribution and the secondgradient distribution; and determines the presence or absence of anabnormality based on the determined distribution divergence and apredetermined distribution divergence threshold set in advance.
 5. Theabnormality detection device according to claim 1, wherein theabnormality determination unit further includes: an output unit foroutputting, in a case that an abnormality is determined by thedetermination of the presence or absence of an abnormality by theabnormality determination unit, an abnormality notification thatincludes an image indicating a region determined to be abnormal.
 6. Theabnormality detection device according to claim 1, wherein: theabnormality detection target object includes a periodic pattern; and thegradient distribution generation unit divides the input image intoregions having a size in which at least one period of the periodicpattern is contained in each region.
 7. An abnormality detection method,the abnormality detection method comprising: a step of inputting aninput image that indicates an abnormality detection target object; astep of dividing the input image into regions of a predetermined sizeand generating, for each region, a gradient distribution that indicatesa distribution of a luminance-gradience direction of the region; a stepof extracting, from among each of the gradient distributions generatedfor each of the regions, a first gradient distribution corresponding toa first region and a second gradient distribution corresponding to asecond region adjacent to the first region; a step of determining adistribution divergence between the first gradient distribution and thesecond gradient distribution by comparing the first gradientdistribution and the second gradient distribution; a step of determininga presence or absence of an abnormality based on the determineddistribution divergence and a predetermined distribution divergencethreshold set in advance; and a step of outputting, in a case that anabnormality is determined by the determination of the presence orabsence of an abnormality, an abnormality notification that includes animage indicating a region determined to be abnormal.
 8. An abnormalitydetection system comprising: a sensor device for image capturing anabnormality detection target object and acquiring an input image thatindicates the abnormality detection target object; a client terminal forrequesting abnormality detection processing with respect to theabnormality detection target object; and an abnormality detection devicefor performing abnormality detection processing with respect to theabnormality detection target object, wherein: the sensor device, theclient terminal, and the abnormality detection device are connected viaa communication network; and the abnormality detection device includes:an image input unit for inputting the input image that indicates theabnormality detection target object from the sensor device; a gradientdistribution generation unit for dividing the input image into regionsof a predetermined size and generating, for each region, a gradientdistribution that indicates a distribution of a luminancegradient-direction of the region; an abnormality determination unit fordetermining a presence or absence of an abnormality by analyzing thegradient distribution generated for each region; and an output unit fortransmitting, in a case that an abnormality is determined by thedetermination of the presence or absence of an abnormality by theabnormality determination unit, an abnormality notification thatincludes an image indicating a region determined to be abnormal to theclient terminal.